Movies are often played to an audience that is not familiar with the original language, and thus cannot understand the sound track of such movies. Two well known common approaches exist to solve this problem. In one approach sub-titles in typed text of the described language are added to the pictures, and the viewers are expected to hear the text in a foreign language and simultaneously to read its translation on the picture itself. Such reading distracts the viewers from the pictures and from the movie in general. Another approach is dubbing, where the original sound-track with the original text is being replaced by another sound-track with the desired language. In this case there is a disturbing mis-match between the sound-track and the movements of the mouth.
There have been some earlier attempts to overcome these disadvantages, none of which have been commercialized because of inherent principal difficulties which made the practical execution unrealistic. Thus, in U.S. Pat. No. 4,600,281 a method is described which performs the measurements of the shape of the mouth manually by a ruler or with a cursor, and corrects the mouth shape by moving pixels within each frame. As will be seen in the description of the invention, the method according to the present invention is inherently different and much superior in the following points: In the present invention the tracking of the shape of the mouth is done automatically and not manually. In the present invention changing the shape of the mouth is done by using a three-dimensional head model, for example like those described by P. Ekman and W. V. Friesen, (Manual for the Facial Action Unit System, Consulting Psychologist Press, Palo Alto 1977). In the present invention the mouth area of the actor is replaced using the mouth area of a reference similarity frame. In the present invention mouth status parameters of the dubber are substituted for mouth status parameters of the actor.
The U.S. Pat. No. 4,260,229 relates to a method of graphically creating lip images. This U.S. patent is totally different from the present invention: In the U.S. patent, speech sounds are analyzed and digitally encoded. In the present invention no sound analysis is done; nor is any required at all.
To make for better viewing of the audio visually dubbed movie, the present invention provides a computerized method wherein, in addition to replacing the sound track to the target text, the mouth movements of the actor are being automatically changed to match the target text. The new mouth movements are linguistly accurate and visually natural looking according to all of the observable parameters of the actor's face.